Thefaithfulnessassumption allows us to infer dependence relations based on d-separation:Pis said to be faithful to graphD when the following holds: Any three variablesX,Y,Z that are not d-separated are conditionally dependent,i.e.,X 6 Y |Z inP.

For adversary's strategy defined in This is the desired result in the lemma. Rearranging the above inequality will yield us desired result. On the other hand, we can also upper bound the above conditional mutual information. Putting together the pieces yields our result. We first prove the result for point error, the result of function error can be achieved by a Jensen's inequality (please see the end of the proof). Convexity is maintained by the maximum operator over two convex functions.

A.1 On the ES fairness notion In this paper, we defined the ES fairness notion as follows, Pr {E Consider classifier R = r (X,A). A.4 Restating Theorem 5 for the statistical parity (SP) fairness notion Here we restate Theorem 5 for the statistical parity. The proof is similar to the proof of Theorem 5. Note that ( X,Y) and A are conditionally independent given A . Pr{r (X, 0) = ˆy |Y = 1,A = 0 } A.7 Numerical Experiment We compared EO and ES fairness notions in Table 2 after adding the following constraints to (13). Next, we prove the second part of the theorem.